Skip to main content

Open Weight Model Evaluation with Integrated Fireworks AI

Dynamo users commonly want to test open weight or custom models with DynamoEval. Fireworks AI is a platform for serving open weight models and custom trained models. This integration doc walks through a Fireworks AI integration example of how Ember-1, Fireworks' post-trained, token-efficient model, can be evaluated by DynamoEval over Fireworks AI's OpenAI-compatible API. The same integration workflow covers any model served on Fireworks.

Fireworks AI connects as a Custom API Language Model. For the request and response formats and every authentication mode, see Custom API Language Models.

Integration Workflow​

Users can start by connecting their Fireworks AI models on the Dynamo Homepage, where they can select Connect AI System, then choose Custom API Endpoints and enter the following details to complete the integration.

Connect AI System dialog with Custom API Endpoints selected, the Fireworks chat completions URL, and Bearer Token authentication

SettingValue
ProviderCustom API Endpoints
API Endpointhttps://api.fireworks.ai/inference/v1/chat/completions
Bearer TokenFIREWORKS_API_KEY, read from the environment
Model IDaccounts/fireworks/models/ember-1
Request TransformationSee below
Response TransformationSee below

Request Transformation​

{
  "model": "accounts/fireworks/models/ember-1",
  "messages": $reduce(messages, $append),
  "max_tokens": seq_len,
  "temperature": temperature
}

Response Transformation​

[choices[0].message.content]

Transformation Code section with the Fireworks request and response JSONata expressions

Running Evaluations Against Your Connected Fireworks Ember-1 Endpoint​

Once the model is connected via the above steps, users can then run model evaluations against Ember-1 (or any Fireworks AI model), by selecting Manage Evaluations. Below are some example insights that can be generated via DynamoEval testing of Ember-1 Fireworks AI models.

  • Model Risk. Red-team Ember-1 across 50+ adversarial tactics against custom policy criteria, including over 100+ out-of-box policy objectives from Dynamo. This includes rapid testing the model's inherent guardrails for safety, prompt injection, EU AI Act Prohibited AI Practices, High Risk Area, US State Law Requirements, and common regulated industry guardrail requirements.
  • Guardrail Impact. Compare catch rate and over-refusal with and without DynamoGuard applied as a guardrail on top of the Fireworks model endpoint.
Last updated on